test(voice): repair and re-enable turn-detection tests in CI - #6715
test(voice): repair and re-enable turn-detection tests in CI#6715brandonbodily wants to merge 1 commit into
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Four test modules are marked `pytest.mark.audio_eot`, covering 83 tests of turn-detection behaviour: tests/test_audio_recognition_turn_detection.py 29 tests/test_audio_turn_detector_fallback.py 40 tests/test_turn_detection_fsm.py 11 tests/test_turn_detection_cloud_stream.py 3 CI runs `make unit-tests` -> `pytest --unit`, which deselects all of them, and no workflow passes `--audio_eot`. None of these 83 tests has been running. test_audio_recognition_turn_detection.py drifted out of sync with the source as a result; 8 of its 29 fail on a clean checkout: - `AudioRecognition.update_turn_detector` -> `_update_turn_detector` - `AudioRecognition.clear_user_turn` -> `_clear_user_turn` - the `_make_full_recognition_for_eou` stub was missing `_turn_backchannel_over_agent` and `_stt_consumer_atask` - the "already flushed" log assertion no longer matched the warning text, which now reads "transcript arrives after turn has been committed" This repairs all four drift points and adds `--audio_eot` to the `unit-tests` target. The other three modules were already green -- they simply never ran. Verified: `pytest --audio_eot` goes from 8 failed/75 passed to 83 passed. `pytest --unit` and `pytest --unit --audio_eot` are identical on every other test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
d0ef2b2 to
3a6664f
Compare
|
Nice — re-enabling these is overdue, and test_late_stt_final_after_flush_short_circuits is exactly the path #6504 and #6415 exercise. I reproduced both on LiveKit Cloud (1.6.5) with audio + support access, linked on those issues — the email-input case ("It's john at" commits via VAD, then "example dot com" arrives after the flush) triggers it deterministically. Once this lands and #6516 fixes the commit path, happy to open the follow-up that adds the on_end_of_turn.assert_not_called() assertion to that test so #6504 stays covered as a regression — you've basically written the blueprint for it above. Let me know if that split works for you. |
3 participants
Summary
Four test modules are marked
pytest.mark.audio_eot, covering 83 tests of turn-detection behaviour:tests/test_audio_recognition_turn_detection.pytests/test_audio_turn_detector_fallback.pytests/test_turn_detection_fsm.pytests/test_turn_detection_cloud_stream.pyCI runs
make unit-tests→pytest --unit, which deselects all of them, and no workflow passes--audio_eot:None of these 83 tests has been running in CI.
Consequence
test_audio_recognition_turn_detection.pydrifted out of sync with the source. On a clean checkout ofmain, 8 of its 29 fail:AudioRecognition.update_turn_detector_update_turn_detector(renamed private)AudioRecognition.clear_user_turn_clear_user_turn(renamed private)_make_full_recognition_for_eoustub_turn_backchannel_over_agent,_stt_consumer_atask"already flushed""transcript arrives after turn has been committed"Each is a one-line repair. The other three modules were already green — they simply never ran.
Changes
--audio_eotto theunit-teststarget so these run alongside--unit.No library code is touched — only the test module and the make target.
Verification
pytest --unitandpytest --unit --audio_eotproduce identical results across every other test in the suite.Why this matters beyond the repair
This is the coverage for turn-detection commit policy, which is where several open correctness issues live (#6504, #3515). With the suite green, adding a single assertion to the existing
test_late_stt_final_after_flush_short_circuitsreproduces #6504 deterministically:That test already drives the exact #6504 scenario — a late STT final after
_turn_detector_flushedis set — and asserts onpredict, onon_eot_prediction, and on the warning, but never on whether a turn is committed. The sibling test two functions below (test_predict_timeout_signals_fallback_and_drops_future) does asserton_end_of_turn.assert_called_once(), so the pattern is established in this file.I've deliberately left that assertion out of this PR — it would fail until #6504 is fixed, and PR #6516 is already open against it. This PR is only the CI repair, so it lands independently and green.
Two other markers look dark, but I didn't touch them
While tracing this I noticed
plugin(26 modules) anddocs(1) are also never selected by any workflow — they're only reachable throughtests/Makefile:unit-tests(--unit --plugin --evals --docs), and CI'smake unit-testsstep runs from the repo root, so it hits the rootmakefiletarget instead.To be clear about what still does work: the plugin matrix isn't idle —
tests/Makefile:testrunspytest --tts, andtest_tts.pyreads thePLUGINenv var (lines 286, 551) to filter its parametrized provider list, so each matrix job genuinely exercises that provider's TTS. What it doesn't run are the 26 dedicatedtest_plugin_*modules, including the*_stt.pyones.I've left those out of this PR on purpose — re-enabling 26 provider-integration modules is a credentials-and-CI-cost decision for maintainers, not something to slip into a test repair. Happy to open a separate issue if that's useful.